In the era of big data, ensuring the quality of datasets has become increasingly crucial across various domains. We propose a comprehensive framework designed to automatically assess and rectify data quality issues in any given dataset, regardless of its specific content, focusing on both textual and numerical data. Our primary objective is to address three fundamental types of defects: absence, redundancy, and incoherence. At the heart of our approach lies a rigorous demand for both explainability and interpretability, ensuring that the rationale behind the identification and correction of data anomalies is transparent and understandable. To achieve this, we adopt a hybrid approach that integrates statistical methods with machine learning algorithms. Indeed, by leveraging statistical techniques alongside machine learning, we strike a balance between accuracy and explainability, enabling users to trust and comprehend the assessment process. Acknowledging the challenges associated with automating the data quality assessment process, particularly in terms of time efficiency and accuracy, we adopt a pragmatic strategy, employing resource-intensive algorithms only when necessary, while favoring simpler, more efficient solutions whenever possible. Through a practical analysis conducted on a publicly provided dataset, we illustrate the challenges that arise when trying to enhance data quality while keeping explainability. We demonstrate the effectiveness of our approach in detecting and rectifying missing values, duplicates and typographical errors as well as the challenges remaining to be addressed to achieve similar accuracy on statistical outliers and logic errors under the constraints set in our work.

The spread of information through social media platforms can create environments possibly hostile to vulnerable communities and silence certain groups in society. To mitigate such instances, several models have been developed to detect hate and offensive speech. Since detecting hate and offensive speech in social media platforms could incorrectly exclude individuals from social media platforms, which can reduce trust, there is a need to create explainable and interpretable models. Thus, we build an explainable and interpretable high performance model based on the XGBoost algorithm, trained on Twitter data. For unbalanced Twitter data, XGboost outperformed the LSTM, AutoGluon, and ULMFiT models on hate speech detection with an F1 score of 0.75 compared to 0.38 and 0.37, and 0.38 respectively. When we down-sampled the data to three separate classes of approximately 5000 tweets, XGBoost performed better than LSTM, AutoGluon, and ULMFiT; with F1 scores for hate speech detection of 0.79 vs 0.69, 0.77, and 0.66 respectively. XGBoost also performed better than LSTM, AutoGluon, and ULMFiT in the down-sampled version for offensive speech detection with F1 score of 0.83 vs 0.88, 0.82, and 0.79 respectively. We use Shapley Additive Explanations (SHAP) on our XGBoost models' outputs to makes it explainable and interpretable compared to LSTM, AutoGluon and ULMFiT that are black-box models.

Strategies based on Explainable Artificial Intelligence - XAI have promoted better human interpretability of the results of black box machine learning models. This sets a precedent for questioning whether or not human expectations are being met when faced with the explanations of this type of model. The XAI measures being currently used (Ciu, Dalex, Eli5, Lofo, Shap, and Skater) provide various forms of explanations, including global rankings of relevance of attributes, which allow for an overview of how the model is explained as a result of its inputs and outputs. These measures provide for an increase in the explainability of the model and a greater interpretability grounded on the context of the problem. Current research points to the need for further studies (within a specific context/problem) on how these explanations meet the Interpretability Expectations of human experts and how they can be used to make the model even more transparent while taking into account specific complexities of the model and dataset being analyzed, as well as important human factors of sensitive real-world contexts/problems. Intending to shed light on the explanations generated by XAI measures and their interpretabilities, this research addresses a real-world classification problem related to homicide prediction, duly endorsed by the scientific community, replicated its proposed black box model and used 6 different XAI measures to generate explanations and 6 different human experts to generate what this research referred to as Interpretability Expectations - IE. The results were computed by means of comparative analysis and identification of relationships among all the attribute ranks produced, and 49% concordance was found among attributes indicated by means of XAI measures and human experts, 41% exclusively by XAI measures and 10% exclusively by human experts. The results allow for answering questions such as: "Do the different XAI measures generate similar explanations for the proposed problem?", "Are the interpretability expectations generated among different human experts similar?","Do the

The need for explainable AI (XAI) is well established but relatively little has been published outside of the supervised learning paradigm. This paper focuses on a principled approach to applying explainability and interpretability to the task of unsupervised anomaly detection. We argue that explainability is principally an algorithmic task and interpretability is principally a cognitive task, and draw on insights from the cognitive sciences to propose a general-purpose method for practical diagnosis using explained anomalies. We define Attribution Error, and demonstrate, using real-world labeled datasets, that our method based on Integrated Gradients (IG) yields significantly lower attribution errors than alternative methods.

In this article, I'll create and train the Deep Neural Network architecture proposed by Wieland Brendel and Matthias Bethge in this research paper. As they state in their paper, the model performance I achieved in this test is similar to the performance of equivalent non-explainable Deep Learning models, and the proposed model, named BagNets, is really able to provide insights into how it achieves its predictions. This tutorial assumes you know the basics of Tensorflow and Keras Layers API. What is an Explainable Deep Learning (DL) Model? There's no single definition, but here and in my daily work, I chose the definition of Holzinger and his contributors, stated in their very enlighted paper.

Senior executives should create a top-down view of how the company will use data, analytics, and AI. This should include a clear statement of the value these tools bring to the organisation, recognition of the associated risks, and clear guidelines and boundaries that can form the basis for more detailed risk-management requirements further down in the organisation. Build on the overarching principles to establish the basic framework for AI risk management. Ensure this covers the full model-development life cycle outlined earlier: ideation, data sourcing, model building and evaluation, industrialisation, and monitoring. Controls should be in place at each stage of the life cycle, so engage early with analytics teams to ensure that the design can be integrated into their existing development approach.

There's widespread consensus that we're in the throes of the fourth industrial revolution Artificial intelligence and its sister technologies is transforming virtually every business. Yet with AI's enormous potential comes great responsibility. The majority (77%) of CEOs say that AI threatens to increase vulnerability and disruption to the ways they do business. Unfortunately, the call for responsible AI has taken a backseat for many companies. Only 25% of companies say that they definitely prioritize considering the ethical implications of an AI solution before investing in it, according to research by PwC.

This book compiles leading research on the development of explainable and interpretable machine learning methods in the context of computer vision and machine learning. Research progress in computer vision and pattern recognition has led to a variety of modeling techniques with almost human-like performance. Although these models have obtained astounding results, they are limited in their explainability and interpretability: what is the rationale behind the decision made? Hence, while good performance is a critical required characteristic for learning machines, explainability and interpretability capabilities are needed to take learning machines to the next step to include them in decision support systems involving human supervision.

The social implications of algorithmic decision-making in sensitive contexts have generated lively debates among multiple stakeholders, suc h as moral and political philosophers, computer scientists, and the public. Yet, the lack of a common language and a conceptual framework for an appropriate bridging of the mor al, technical, and political aspects of the debate prevents the discussion to be as effective a s it can be. Social scientists and psychologists are contributing to this debate by gather ing a wealth of empirical data, yet a philosophical analysis of the social implications of a lgorithmic decision-making remains comparatively impoverished. In attempting to address this lacuna, this paper argues that a hierarchy of different types of explanations for why and how an algorithmic decision outcome is achieved can establish the relevant connection between t he moral and technical aspects of algorithmic decision-making. In particular, I offer a multifaceted conceptual framework for the explanations and the interpretations of algorithmic de cisions, and I claim that this framework can lay the groundwork for a focused discussion among mu ltiple stakeholders about the social implications of algorithmic decision-making, as we ll as AI governance and ethics more generally.

It doesn't take a data scientist to work out that the machine and deep learning algorithms built into automation and artificial intelligence systems lack transparency. It also doesn't take a great deal of detective work to see that many of these systems contain an imprint of the unconscious biases of the engineers that helped to develop them. Arguably, in the midst of what The Economist termed a techlash, this lack of transparency has only (ironically) become more visible. While many of the incidents that have contributed towards the techlash are as much issues caused by a mixture of corporate self-interest and an alarming absence of governance and accountability, there's no escaping the fact that the practice of data science and machine learning engineering naturally find their way hooked onto some of the year's biggest business and politics stories. It's in this context that the concepts of explainability and interpretability have taken on new urgency.